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TITLE OF THE INVENTION 

METHOD AND APPARATUS FOR 
AUTOMATIC LOCAL PATH PLANNING FOR VIRTUAL COLONOSCOPY. 

5 

CROSS-REFERENCE TO RELATED APPLICATIONS 

PRIORITY 

Specific reference is hereby made to copending U.S. Provisional Patent Application 
10 No. 60/443,734, filed in the United States Patent and Trademark Office on January 30, 2003 in 
the name of inventor Bernhard Geiger, and entitled AUTOMATIC LOCAL PATH PLANNING 
FOR VIRTUAL COLONOSCOPY, whereof the disclosure is hereby incorporated by reference 
herein, and whereof the benefit of priority is claimed. 

Specific reference is also hereby made to copending U.S. Provisional Patent Application 
15 No. 60/470,579, filed in the United States Patent and Trademark Office on May 14, 2003 in the 
name of inventors Bernhard Geiger and Jean-Daniel Boissonnat, and entitled FAST 
CENTERLINE EXTRACTION, whereof the disclosure is hereby incorporated by reference 
herein, and whereof the benefit of priority is claimed. 

The present application relates generally to computer vision and imaging systems and, more 
20 particularly, to a system and method for automatic local path planning such as may be utilized 
for virtual endoscopy and virtual colonoscopy. 

BACKGROUND OF THE INVENTION 

Virtual colonoscopy (VC) refers to a method of diagnosis based on computer simulation of 

25 standard, minimally invasive endoscopic procedures using patient specific three-dimensional 

(3D) anatomic data sets. Examples of current endoscopic procedures include bronchoscopy, 

sinusoscopy, upper gastro-intestinal endoscopy, colonoscopy, cystoscopy, cardioscopy, and 

urethroscopy. VC visualization of non-invasively obtained patient specific anatomic structures 

avoids risks, such as perforation, infection, hemorrhage, and so forth, associated with real 

30 endoscopy, and provides the endoscopist with important information prior to performing an 
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actual endoscopic examination. Such understanding can minimize procedural difficulties, 
decrease patient morbidity, enhance training and foster a better understanding of therapeutic 
results. 

In virtual colonoscopy, 3D images are created from two-dimensional (2D) computerized 
5 tomography (CT) or magnetic resonance (MR) data, for example, by volume rendering. 

These 3D images are created to simulate images coming from an actual endoscope, such as a 
fiber optic endoscope. This means that a viewpoint of the virtual endoscope has to be chosen 
inside a lumen of the organ or other human structure, and with a wide angle of view, typically 
about 100 degrees of arc. This viewpoint has to move along the inside of the lumen , which 
10 means that a 3D translation and a 3D rotation have to be applied. Controlling these parameters 
interactively is a challenge. 

A commonly used technique for navigating a viewpoint of a virtual endoscope is to calculate a 
"flight path" beforehand and automatically move the viewpoint of the virtual colonoscope along 
this path. However, this technique requires a segmentation and trajectory calculation step that is 
1 5 time-consuming and can fail. 

Applicant's prior patent application No. 10/322,326, filed December 18, 2002, and entitled 
AUTOMATIC NAVIGATION FOR VIRTUAL ENDOSCOPY and whereof the disclosure is 
hereby incorporated herein by reference to the extent it is not incompatible with the present 
invention, discloses a system and method of navigation inside a colon dataset, using the longest 

20 view ray to advance. The aforesaid prior patent application shows a system and method for 
automatic navigation of a viewpoint of an endoscope in virtual colonoscopy is provided. The 
system determines automatically a direction and orientation of a virtual endoscope. Therefore a 
user needs to control only one parameter - forward or backward speed and the method allows 
immediate interactive navigation inside an organ without preprocessing, e.g. segmentation and 

25 path generation. The method shown in the said application includes the steps of (a) determining 
an initial viewpoint of the virtual endoscope, the initial viewpoint having a first center point and 
a first direction; (b) determining a longest ray from the initial viewpoint to the lumen, the longest 
ray having a first longest ray direction; (c) determining a second direction between the first 
direction of the initial viewpoint and the first longest ray direction; (d) turning the viewpoint to 

30 the second direction and moving the initial viewpoint a first predetermined distance in a first 
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direction of the initial viewpoint; (e) calculating a second center point of the viewpoint; (f) . 
moving the viewpoint to the second center point; and repeating steps (b) through (f) until the 
viewpoint reaches an intended target. 



5 BRIEF SUMMARY OF THE INVENTION 

It is herein recognized that it is nevertheless possible, under certain circumstances, for operation 
of the system disclosed in the aforesaid prior patent application (Attorney Docket No. 
2001P24461US), to get stuck in a sharp bend, or a deep Haustral fold as may occur in virtual 
colonoscopy. 

1 0 According to one aspect of the present invention, a method for navigating a viewpoint of a 
virtual colonoscope in a lumen of a structure is provided. In accordance with an aspect of the 
present invention, a method for automatic path planning comprises the following steps: defining 
a sub volume around the current endoscope position; performing a region growing inside the 
lumen, starting from the endoscope position; calculating and clustering the intersection of the 

1 5 grown region with the 6 faces of a cube circumscribing the region of interest; calculating 
approximated centerlines from the endoscope position to the center of each of the clusters; 
comparing each path with the current path of the endoscope; and finding the best score. 

In accordance with another aspect of the invention, a method for automatic local path planning 
for a virtual endoscope, comprises the steps of: deriving a colon dataset obtained by a 

20 colonoscopy protocol for utilization in subsequent steps; defining a sub volume around a current 
endoscope position in a lumen; performing a region growing inside the lumen, starting from the 
current endoscope position; calculating and clustering the intersection of the region with the 
faces of a cube circumscribing the sub volume; calculating approximated centerline paths from 
the current endoscope position to the center of each cluster formed in the preceding step; 

25 comparing each of the centerline paths with a current path exhibited by the endoscope; assigning 
a score based on the comparing, to each of the centerline paths; and selecting a centerline path 
based on the score. 

In accordance with another aspect of the invention, the step of deriving a colon dataset obtained 
by a colonoscopy protocol comprises deriving the data set by computerized tomography (CT). 
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In accordance with another aspect of the invention, the step of deriving a colon dataset obtained 
by a colonoscopy protocol comprises deriving the data set by magnetic resonance (MR). 

In accordance with another aspect of the invention, a method for automatic local path planning 
for a virtual endoscope includes a step of defining the cube to be of a given number of voxels 
5 centered around the current endoscope. 

In accordance with another aspect of the invention, a step of calculating approximated centerline 
paths comprises: calculating an initial path; and centering and smoothing the initial path. 

In accordance with another aspect of the invention, a step of centering and smoothing the initial 
path comprises using Gaussian smoothing. 

10 In accordance with another aspect of the invention, a step of centering and smoothing the initial 
path comprises: setting a sphere at a vertex location; increasing the size of the sphere size until it 
comes into collision with a wall of the lumen; calculating a translation force from the collision; 
applying the translation force until the sphere is no longer in collision; increasing the size of the 
sphere and it again comes into further collision with the wall; calculating a further translation 

15 force; applying the further translation force until the sphere is no longer in collision; repeating 
the foregoing three steps until the sphere reaches a final position where it cannot grow any more 
without collision; indicating the final position as a final vertex position. 

In accordance with another aspect of the invention, a method for automatic local path planning 
for a virtual endoscope, comprises the steps of: deriving a colon dataset obtained by a 

20 colonoscopy protocol for utilization in subsequent steps; defining a sub volume around a current 
endoscope position in a lumen; performing a region growing inside the lumen, starting from the 
current endoscope position; calculating and clustering the intersection of the region with the 
faces of a cube circumscribing the sub volume; calculating approximated centerline paths from 
the current endoscope position to the center of each cluster formed in the preceding step by: 

25 calculating an initial path; centering and smoothing the initial path; comparing each of the 

centerline paths with a current path exhibited by the endoscope; assigning a score based on the 
comparing, to each of the centerline paths; and selecting a centerline path based on the score. 

In accordance with another aspect of the invention, a method for automatic local path planning 
for a virtual endoscope, comprising the steps of: defining a sub volume around a current 
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endoscope position in a lumen; performing a region growing inside the lumen, starting from the 
current endoscope position; calculating and clustering the intersection of the region with the 
faces of a cube circumscribing the sub volume; calculating approximated centerline paths from 
the current endoscope position to the center of each cluster formed in the preceding step; 
5 comparing each of the centerline paths with a current path exhibited by the endoscope; and 
selecting an optimal centerline path based on the comparing. 

In accordance with another aspect of the invention, a system for automatic local path planning 
for a virtual endoscope, comprises: apparatus for defining a sub volume around a current 
endoscope position in a lumen; apparatus for performing a region growing inside the lumen, 

10 starting from the current endoscope position; apparatus for calculating and clustering the 

intersection of the region with the faces of a cube circumscribing the sub volume; apparatus for 
calculating approximated centerline paths from the current endoscope position to the center of 
each cluster formed in the preceding step; apparatus for comparing each of the centerline paths 
with a current path exhibited by the endoscope; and apparatus for selecting an optimal centerline 

1 5 path based on the comparing. 

In accordance with another aspect of the invention, a system for automatic local path planning 
for a virtual endoscope, comprises: apparatus for deriving a colon dataset obtained by a 
colonoscopy protocol for utilization in subsequent steps; apparatus for defining a sub volume 
around a current endoscope position in a lumen; apparatus for performing a region growing 

20 inside the lumen, starting from the current endoscope position; apparatus for calculating and 
clustering the intersection of the region with the faces of a cube circumscribing the sub volume; 
apparatus for calculating approximated centerline paths from the current endoscope position to 
the center of each cluster formed in the preceding step; apparatus for comparing each of the 
centerline paths with a current path exhibited by the endoscope; apparatus for assigning a score 

25 based on the comparing, to each of the centerline paths; and apparatus for selecting a centerline 
path based on the score. 

In accordance with another aspect of the invention, apparatus for centering and smoothing the 
initial path comprises apparatus for: setting a sphere at a vertex location; increasing the size of 
the sphere size until it comes into collision with a wall of the lumen; calculating a translation 
30 force from the collision; applying the translation force until the sphere is no longer in collision; 
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increasing the size of the sphere and it again comes into further collision with the wall; 
calculating a further translation force; applying the further translation force until the sphere is no 
longer in collision; repeating the foregoing three steps until the sphere reaches a final position 
where it cannot grow any more without collision; indicating the final position as a final vertex 
position. 

In accordance with another aspect of the invention, a method for automatic local path planning 
for a virtual endoscope comprises the steps of defining a sub volume around a current endoscope 
position in a lumen; performing a region growing inside the lumen, starting from the current 
endoscope position; calculating and clustering the intersection of the region with the faces of a 
cube circumscribing the sub volume; calculating approximated centerline paths from the current 
endoscope position to the center of each cluster formed in the preceding step; comparing each of 
the centerline paths with a current path exhibited by the endoscope; and selecting an optimal 
centerline path based on the comparison. 

1 5 BRIEF DESCRIPTION OF THE SEVERAL VIEWS OF THE DRAWINGS 

These and other aspects of the present invention will be more fully understood from the detailed 
description which follows, in conjunction with the drawing, in which 

Figure 1 shows diagrammatically a typical situation, helpful to a fuller understanding of the 
present invention, wherein the progress of an endoscope is stuck in a sharp bend; 

20 Figures 2 and 3 show voxel paths relating to centerline extraction as may be utilized in 
embodiments of the present invention; and 

Figure 4 shows a centering step as may be utilized in embodiments of the present invention. 

DETAILED DESCRIPTION OF THE INVENTION 

25 It will be understood that the method and system of the present invention are implemented 
utilizing a programmable digital computer and that the operations herein described are in 
reference to such an implementation. In the context of imaging, terms such as "air", "lumen", 
etc. are typically intended to refer to the corresponding imaging of these features. 
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As hereinbefore explained, it is possible, under certain circumstances, for operation of prior-art 
systems for virtual colonoscopy to get stuck in a sharp bend, or a deep Haustral fold as may 
occur in virtual colonoscopy. Figure 1 shows a typical situation where the endoscope is stuck 
in a sharp bend. 

5 In accordance with embodiment of the present invention, a region grown from the endoscope 
position intersects the cube in three clusters, cO, cl and c2. The system calculates new 
centerlines from the endoscope towards the center of each cluster and compares each path with 
the path of the endoscope. A score is calculated, which reflects the dissemblance of the path 
with the incoming path. The path with the highest score is most likely the one that continues in 
1 0 the direction of the colon. If there is only one cluster, the system decides that there is a block 
and that it has reached a dead end. 

In accordance with an embodiment of the present invention, a method for automatic path 
planning includes a step of defining a sub volume around the current endoscope position. Next, 
a step of performing a region growing inside the lumen is carried out, starting from the 
1 5 endoscope position. This is followed by the steps of calculating and clustering the intersection 
of the grown region with the 6 faces of the cube circumscribing the region of interest. 
Approximated centerlines from the endoscope position to the center of each of the clusters are 
then calculated. Each path is compared with the current path of the endoscope and a best score is 
found. 

20 The step of calculating approximated centerlines may be performed in any of various ways. A 
number of centerline algorithms use thinning, morphological operators, distance transforms, 
minimum cost paths, Dijkstra's algorithm, etc. Background material may be found in the 
literature, for example: 

Zhou et al., "Three-Dimensional Skeleton and Centerline Generation Based on an Approximate 
25 Minimum Distance Field," The Visual Computer, 14:303-314 (1998); R. Truyen, T. Deschamps, 
L.D. Cohen, "Clinical evaluation of an automatic path tracker for virtual colonoscopy, Medical 
Image Computing and Computer-Assisted Intervention (MICCAI), Utrecht, Netherlands, 
October 2001; Chen et al., "A Fast Algorithm to Generate Centerline for Virtual Colonoscopy," 
SPIE Conference, Feb. 12-18, 2000; Richard Robb, "Virtual (Computed) Endoscopy: 
30 Development and Evaluation Using the Visible Human Datasets," Oct. 7-8, 1996, 
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www.mayo.edu; U.S. Patent 6,5 14,082 entitled "System and method for performing a three- 
dimensional examination with collapse correction. 

Another way of carrying out colon segmentation comprises performing a start- and endpoint 
calculation, and performing an initial path calculation as described in the afore-mentioned 
5 provisional patent application No. 60/470,579 by Geiger et al. This is followed by path centering 
and smoothing. This starts typically with a colon dataset that has been obtained using a 
colonoscopy protocol typically including bowel preparation and air insufflation. The dataset is 
segmented by applying a threshold to air and doing connected component analysis, whereby 
connected components that do not belong to the colon are discarded, either automatically or by 

10 manual selection. Starting from a first voxel that belongs to the colon, distance labeling of 
voxels is carried out, typically with consecutive numbering, the first voxel getting label 0, its 
neighbors label 1, their neighbors label 2, and so forth. A search is performed for the highest 
label, this becoming the starting point pO. From pO, a new distance label map is created by 
repeating the prior step of distance labeling and obtaining another voxel with the highest number. 

1 5 This is the endpoint p 1 . 

From pi, distance labels are used to get a path of connected voxels that ends in pO. This is done 
by searching among the neighbors of p2 for a voxel with a smaller label, and so forth, until pO is 
reached. See Figure 2a and Figure 3a which show the initial voxel path. The resulting path is 
generally jagged and is smoothed by using known techniques, for example, Gaussian smoothing. 

20 Any vertex is replaced by the weighted average of its n neighbors, and the process is repeated 
over a number of iterations. Any new vertex position will be tested for collision with the colon 
wall by verifying whether the new coordinate still lies within the segmented colon. In the event 
of a collision, the vertex is left at the last collision-free position. In a sense, this process may be 
likened to the path of a stretched flexible weightless string passing through the colon. See Figure 

25 2b and Figure 3b which show the initial smoothing step. Figures 2(c) and 3 (c) show the final 
centering. 

In accordance with a described embodiment, this smooth path is centered using spheres with 
increasing sizes. See Figure 4. A small sphere is centered at a vertex along the path. The 
vertices on this sphere are checked for collision with the colon wall. If the vertices are in a 
30 collision, a force is defined based on the sphere normals. This force is used to move the sphere 
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away from the wall. The sphere is constrained to move on a plane perpendicular to the path. If 
the sphere is no longer in collision, the size of the sphere is increased and the collision 
calculation and shift is repeated, the process stopping when the sphere cannot be further shifted 
and/or increased in size without creating a collision. Te center of the sphere is now taken as the 
5 new position for the vertex. The process repeats for the next vertex of the trajectory or path. See 
Figure 1c. A description of the collision detection technique and calculation of translation force 
is given in a publication by Geiger, B., "Real-Time Collision Detection and Response for 
Complex Environments," Computer Graphics International 2000; June 19-23, 2000; Geneva, 
Switzerland. However, the collision detection and force calculation herein is done directly on 
10 the voxel, rather than on polyhedral reconstructions, although it generally follows the ideas 
outlined in the aforementioned paper by Geiger. Thereafter, and after centering, the path 
undergoes another Gaussian smoothing with collision control, this time using fewer iterations 
and a smaller neighborhood. 

Figure 4 shows the summarized procedure for the centering step: a sphere is set at the vertex 
15 location (a); the sphere size is increased until it collides with the wall. From the collision, a 
translation force is calculated (b). The translation is applied until the sphere is no longer in 
collision (c) . The size of the sphere is again increased and it is now colliding with the wall. A 
translation is calculated (d). After translation, the sphere reaches a position where it cannot grow 
any more. This represents the final vertex position (c). 

20 A preferred way, in accordance with the principles of the present invention and which does not 
use such morphological operations, is faster in certain situations. 

In accordance with an embodiment of the invention, a method for automatic local path planning 
for virtual colonoscopy comprises the following steps. 

Automatic segmentation is performed wherein the automatic segmentation cuts out a block of a 
25 selected cube size, (for example 128x128x128) voxels centered at the position of the endoscope. 
Starting with the voxel containing the endoscope as a seed, a region growing inside air is 
performed. All voxels containing air and belonging to the same lumen are labeled. This air 
column intersects the cube at various locations. These locations are clustered, whereby voxels 
that share faces are added to the same cluster, and the center of each cluster is calculated by 
30 averaging the voxel coordinates of voxels of each cluster. Centerlines are calculated from the 
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center of each cluster to the endoscope position. Each centerline is compared to the path of the 
endoscope. Starting from the endoscope position, the absolute value of the difference of points 
on a centerline is added to the corresponding point on the endoscope path. The sum is divided 
by the length to provide a score. The higher this score, the more distant is the centerline from the 
5 incoming path. The path with the highest score is the best candidate to follow. If there is only 
one path, it is most likely the one that the endoscope moved in, and the system knows there is a 
dead end. 

It will be understood that the invention, which has been described primarily in the context of 
virtual colonoscopy in which it is most useful, is nevertheless also applicable to various other 
1 0 virtual endoscopic type examinations. While the invention has been described by way of 

exemplary embodiments, it will also be understood by one of skill in the art to which it pertains 
that various changes and modifications can be made which do not depart from the spirit of the 
invention which is defined by the claims following. 
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